Robots.txtThere are many search engines that use web crawlers with sophisticated algorithms to index the content of your site. These crawlers can sometimes find content that you do not want indexed. You can add a robots.txt file to your KommerceServer site to tell search engines about content that you want to exclude. KommerceServer includes a robots.txt file in its standard installation. A small excerpt of the file is shown below:
If you find search engines are indexing files in your site that you don’t want, you can modify this file. There are numerous references on the internet with information about robots.txt. For example, Google provides a lot of information in the maintenance section of its support site: https://support.google.com/webmasters/answer/6062608?hl=en&ref_topic=6061961&rd=1. |